CS 598 : Theoretical Machine Learning

We have already seen in previous lectures that a concept class C is PAC-learnable by the hypothesis class H if there exists an algorithm A such that for all c ∈ C, for all distributions D over X, for all > 0, for all δ > 0, A takes m = finite number of examples given in the form S = 〈 (x1, c(x1)), ..., (xm, c(xm)) 〉 , where each xi chosen from the space X at random according to the target distribution D, and produces a hypothesis h ∈ H such that Pr[errD(h) ≤ ] ≥ 1− δ After going through the definition, there are few questions which can be asked, such as what if we don’t have an algorithm that works for every and every δ. For instance, what if we have an algorithm that only works for δ = 1 3 . This algorithm guarantees a success probability of 23 . Technically this algorithm is not a PAC-learning algorithm because one cannot give any success probability as input and expect to achieve success with that probability. Can we use this algorithm which has a success probability of 23 to get arbitrary success probability?. One possible way is to run the algorithm many times, and we know from Chernoff bounds that the algorithm will succeed with probability ≥ 1−δ. In particular, if it is run for log(1δ times , it can be guaranteed that in one of the tries, it could output a good hypothesis. Lets look at another question, suppose there is an algorithm that doesn’t achieve any value of (error of the algorithm) i.e it cannot achieve any arbitrary error. For instance, say we have an algorithm that only gets = 0.49. In binary classification, if one were to just randomly guess then the error achieved would be half. Now, this algorithm performs slightly better than random guessing with error rate 0.49. Can we use this algorithm to achieve arbitrary accuracy?. This algorithm described above is known as Weak PAC-learning. For all distributions D ∼ x, for all δ > 0 the algorithm A uses finite data and produces a hypothesis h ∈ H such that Pr[errD(h) ≤ 12 − γ] ≥ 1 − δ, for some fixed γ > 0. This learning does better than random guessing and is a easy to design compared to some algorithm that does very well on entire instance space(X) which gets error as close to 0(harder problem to design). It is clear from previous argument, strong PAC learning implies weak PAC learning. But can we say weak learning implies strong learning?. In fact, we will see in the following section, it is true. We will show that by combining these weak learning in some specific way we can achieve a strong learner. Lets see an example,where the hypothesis class(H) is a class of boolean disjunctions.We have n boolean variables(x1, x2, ..., xn) and the target function is some disjunction f ∗ = (x1 ∨ x2 ∨ ... ∨ xk). And we are trying to learn this disjunction.